Summary

Replaces BBOT's entire HTTP infrastructure with blasthttp, a Rust-based HTTP library with Python bindings. This eliminates the httpx Go binary subprocess, the curl subprocess helper, and the HTTPEngine ZMQ subprocess — all HTTP now runs in-process through a shared blasthttp client.

What changed

New HTTP engine:

• All HTTP requests go through helpers.request() → WebHelper → shared blasthttp.BlastHTTP() client
• Rate limiting via web.http_rate_limit config, enforced at the client level across all callers
• resolve_ip parameter for DNS pinning (like curl --resolve) — connects to a specific IP while preserving hostname for Host header and TLS SNI
• request_target parameter for request-line override (SSRF/smuggling testing)
• TLS certificate info (CN, SANs, issuer) available on every HTTPS response via response.cert_info

Shared event loop (blasthttp 0.2.0):

• blasthttp upgraded to 0.2.0 which uses pyo3-async-runtimes to return native Python coroutines via future_into_py()
• All run_in_executor_io() wrappers around blasthttp calls replaced with direct await — HTTP requests no longer consume OS threads
• IO thread pool shrunk (only remaining caller is wafw00f)
• Thread pool backlog status line removed (no longer relevant)

Removed:

• httpx Go binary module and its test (bbot/modules/httpx.py)
• ffuf Go binary module and its test (bbot/modules/ffuf.py, ffuf_shortnames.py)
• HTTPEngine ZMQ subprocess (bbot/core/helpers/web/engine.py)
• AsyncClient / httpx Python library (bbot/core/helpers/web/client.py)
• helpers.web.curl() subprocess helper
• DEP_FFUF and DEP_CURL shared dependency definitions
• httpx Python library from dependencies

Added:

• bbot/modules/http.py — native HTTP module using blasthttp batch API (replaces httpx Go binary)
• bbot/modules/web_brute.py — native web fuzzer using blasthttp batch API (replaces ffuf)
• bbot/modules/web_brute_shortnames.py — IIS shortname resolver using ML prediction
• bbot/modules/generic_ssrf.py — SSRF detection module
• bbot/modules/output/webhook.py — renamed from output/http.py to avoid collision with scan module
• bbot/core/helpers/web/blast_response.py — response wrapper for blasthttp PyO3 objects
• bbot/test/mock_blasthttp.py — mock infrastructure for test HTTP interception
• Rate limit test (test_web_rate_limit.py)
• Download timeout default (5 minutes) for large wordlist files

Updated:

• sslcert module rewritten to use blasthttp cert_info instead of independent pyOpenSSL connections
• host_header and generic_ssrf modules converted from curl() to request() with resolve_ip/request_target
• elastic output module fixed to import from webhook instead of deleted http
• All module tests updated for blasthttp mock API (blasthttp_mock.add_response())
• Presets updated: dirbust-light, dirbust-heavy, dotnet-audit reference web_brute instead of ffuf
• Test mock conftest passes through resolve_ip=127.0.0.1 requests to real blasthttp

Bug fixes

• Blacklist.get() NoneType crash — _make_event_seed() could return None when host validation fails, causing AttributeError on .host access. Events hitting this path were silently dropped from the scan pipeline. Added None guard with defensive tests.
• DNS CNAME escaped quotes — clean_dns_record() didn't strip quote characters that dnspython's to_text() can produce on certain record types, causing ValidationError and silently skipping DNS children. Added .strip("'\"") before rstrip(".").
• Certspotter rate-limit crash — API returns a JSON dict (not list) when rate-limited, causing AttributeError: 'str' object has no attribute 'get' when iterating dict keys. Added isinstance(json, list) guard.
• Excavate YARA blocking event loop — yara_rules.match() was called synchronously on the event loop, serializing all 8 excavate workers despite _module_threads = 8. Offloaded to run_in_executor_cpu() for real parallelism (YARA releases the GIL). ~2.5-3x throughput improvement.
• Stale ip-* and http-title-* tags — The new http module was creating ip-{hostname} and http-title-{title} tags instead of using _resolved_hosts and http_title like the old httpx module did. Fixed to match post-naming-standardization conventions from PR Preset naming standardization / tag cleanup #2986.
• _drain_queues() infinite loop on Ctrl+C — When a module's queue was None/False, the drain loop never raised QueueEmpty and spun forever, causing the scan to hang after "Aborting scan" until a second Ctrl+C. Moved the None/False check outside the loop.
• web_brute sequential bottleneck — _module_threads defaulted to 1, meaning queued URLs were fuzzed one at a time. Bumped to 4. Added configurable concurrency option. Wired the previously unused rate option through to blasthttp request_batch.
• blasthttp rate limit override — When both a global rate limit and a per-call rate limit were set, the global always won unconditionally. Fixed in blasthttp 0.1.4 (blacklanternsecurity/blasthttp#12) to use min(global, per_call) so modules can enforce tighter limits.

Dependency changes

• Added: blasthttp>=0.2.0
• Removed: httpx>=0.28.1

Merge from dev + baddns 2.2.0

Merged dev into this branch and resolved the baddns conflicts by bumping to baddns 2.2.0, which ships the blasthttp-based rewrite:

• Upstream baddns now takes an http_client instance (renamed from http_client_class) and defaults to a shared BlastHTTP() when none is provided
• Baddns internals (HttpManager, mtasts, references, matcher, etc.) fully converted off httpx: response.status/response.body attribute shapes, list-of-tuple headers normalized via a new headers_to_dict() helper
• bbot/modules/baddns.py and baddns_direct.py now pass http_client=self.helpers.blasthttp and dns_client=self.scan.helpers.dns.blastdns — baddns submodules reuse the same rate-limited shared client as the rest of bbot
• Dropped baddns~=2.1.0 → baddns~=2.2.0 in module deps_pip and pyproject.toml

Additional bug fix

• portscan ping_first mode losing hostname correlation — emit_open_port() creates a fresh DNS_NAME event for each alive host, which is immediately fed into a second make_targets() call for the SYN scan. Because dnsresolve runs asynchronously after emit_event, that new event's resolved_hosts was empty when the SYN correlator read it, so the hostname silently dropped out of the correlator. Ports found on shared IPs then propagated only to the IP-range parent — OPEN_TCP_PORT events for the hostnames were intermittently missing. Fixed by seeding event._resolved_hosts from the parent DNS_NAME at creation time.

Post-merge cleanup

• Restore _task_counter on precheck/postcheck without re-introducing event-loop saturation — The earlier saturation fix had stripped _task_counter.count(...) wrappers from precheck/postcheck on the per-event hot path, but those wrappers are load-bearing for finished-detection (is_finished reads _task_counter.value > 0). Without them a module can be marked finished while events are still mid-check. Removed the asyncio.Lock from TaskCounter instead — the lock had no real synchronization role on the single-threaded event loop, just two forced event-loop yields per count() call — then re-added the wrappers in _events_waiting and BaseInterceptModule._worker. Hot-path cost is now a uuid + dict insert/pop with no event-loop yield. (bbot/core/helpers/async_helpers.py, bbot/modules/base.py)

• Remove vestigial _ensure_minimal_target shim — Originally added so WebHelper.__init__ (which reads preset.target at construction) could be safely accessed before Preset.bake() ran. That pre-bake helpers path lives on the asn-as-targets / scope-rework branch where Scanner.helpers falls back to self._unbaked_preset.helpers when self.preset is None. On this branch Scanner.__init__ synchronously bakes the preset, so by the time anyone reads .helpers the target already exists. Verified the shim never fires by adding a diagnostic raise inside it and running step_1 + sample module tests + bbot --help/--list-modules.

• Remove BBOTTarget pickle support — BaseTarget.__getstate__/__setstate__ and ScanBlacklist.__setstate__ existed solely so BBOTTarget could be passed to the HTTPEngine ZMQ subprocess via server_kwargs. That subprocess is gone — blasthttp runs in-process — so the workaround for RadixTarget (PyO3) being un-picklable is no longer needed. Also dropped the test_target_pickle round-trip test.

• Restore compact form of test_manager_scope_accuracy.py and harden the ruff exemption — The file is excluded from ruff format via pyproject.toml's format.exclude, but explicit-file invocations (pre-commit hooks, ruff format <file>) bypass the exclude unless force-exclude = true is also set. The file had been re-expanded from ~900 to 3209 lines as a side-effect. Re-compacted via ruff format --line-length=250 --config 'format.skip-magic-trailing-comma=true' (down to 871 lines) and added force-exclude = true so the existing exclude is honored on explicit invocations too.